{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/tomasonjo/blogs/blob/master/Marvel_series/Exploratory%20graph%20analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iXLxc35asnnV"
   },
   "source": [
    "* Updated to GDS 2.0 version\n",
    "* Link to original blog post: https://towardsdatascience.com/exploratory-network-analysis-of-marvel-universe-c557f4959048"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a51ym4p3stDD",
    "outputId": "8983fc4d-63f1-4537-ea41-c9c146601ee5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting neo4j\n",
      "  Downloading neo4j-4.4.2.tar.gz (89 kB)\n",
      "\u001b[?25l\r",
      "\u001b[K     |███▋                            | 10 kB 19.9 MB/s eta 0:00:01\r",
      "\u001b[K     |███████▎                        | 20 kB 13.6 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████                     | 30 kB 10.5 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████▋                 | 40 kB 9.2 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████▎             | 51 kB 4.3 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████          | 61 kB 5.0 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 71 kB 5.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 81 kB 6.1 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 89 kB 4.0 MB/s \n",
      "\u001b[?25hRequirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from neo4j) (2018.9)\n",
      "Building wheels for collected packages: neo4j\n",
      "  Building wheel for neo4j (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for neo4j: filename=neo4j-4.4.2-py3-none-any.whl size=115365 sha256=37440807321d6c4c73bd85c38c1683cf854f8a8e68d3ed332d5782abd497de89\n",
      "  Stored in directory: /root/.cache/pip/wheels/10/d6/28/95029d7f69690dbc3b93e4933197357987de34fbd44b50a0e4\n",
      "Successfully built neo4j\n",
      "Installing collected packages: neo4j\n",
      "Successfully installed neo4j-4.4.2\n"
     ]
    }
   ],
   "source": [
    "!pip install neo4j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_-yzSIlvswPi"
   },
   "source": [
    "I recommend you setup a [blank project on Neo4j Sandbox environment](https://sandbox.neo4j.com/?usecase=blank-sandbox), but you can also use other environment versions\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "uIStX_HGSsH-"
   },
   "outputs": [],
   "source": [
    "# Define Neo4j connections\n",
    "import pandas as pd\n",
    "from neo4j import GraphDatabase\n",
    "\n",
    "host = 'bolt://44.193.28.203:7687'\n",
    "user = 'neo4j'\n",
    "password = 'combatants-coordinates-tugs'\n",
    "driver = GraphDatabase.driver(host,auth=(user, password))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "I3C3cMuiSsIC"
   },
   "outputs": [],
   "source": [
    "def run_query(query):\n",
    "    with driver.session() as session:\n",
    "        result = session.run(query)\n",
    "        return pd.DataFrame([r.values() for r in result], columns=result.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "vVOKmHldSsIC"
   },
   "outputs": [],
   "source": [
    "import_queries = \"\"\"\n",
    "\n",
    "CALL apoc.schema.assert({Character:['name']},{Comic:['id'], Character:['id'], Event:['id'], Group:['id']});\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroes.csv\" as row\n",
    "CREATE (c:Character)\n",
    "SET c += row;\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/groups.csv\" as row\n",
    "CREATE (c:Group)\n",
    "SET c += row;\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/events.csv\" as row\n",
    "CREATE (c:Event)\n",
    "SET c += row;\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/comics.csv\" as row\n",
    "CREATE (c:Comic)\n",
    "SET c += apoc.map.clean(row,[],[\"null\"]);\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroToComics.csv\" as row\n",
    "MATCH (c:Character{id:row.hero})\n",
    "MATCH (co:Comic{id:row.comic})\n",
    "MERGE (c)-[:APPEARED_IN]->(co);\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroToEvent.csv\" as row\n",
    "MATCH (c:Character{id:row.hero})\n",
    "MATCH (e:Event{id:row.event})\n",
    "MERGE (c)-[:PART_OF_EVENT]->(e);\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroToGroup.csv\" as row\n",
    "MATCH (c:Character{id:row.hero})\n",
    "MATCH (g:Group{id:row.group})\n",
    "MERGE (c)-[:PART_OF_GROUP]->(g);\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroToHero.csv\" as row\n",
    "MATCH (s:Character{id:row.source})\n",
    "MATCH (t:Character{id:row.target})\n",
    "CALL apoc.create.relationship(s,row.type, {}, t) YIELD rel\n",
    "RETURN distinct 'done';\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroStats.csv\" as row\n",
    "MATCH (s:Character{id:row.hero})\n",
    "CREATE (s)-[:HAS_STATS]->(stats:Stats)\n",
    "SET stats += apoc.map.clean(row,['hero'],[]);\n",
    "\n",
    "LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/tomasonjo/blog-datasets/main/Marvel/heroFlight.csv\" as row\n",
    "MATCH (s:Character{id:row.hero})\n",
    "SET s.flight = row.flight;\n",
    "\n",
    "MATCH (s:Stats)\n",
    "WITH keys(s) as keys LIMIT 1\n",
    "MATCH (s:Stats)\n",
    "UNWIND keys as key\n",
    "CALL apoc.create.setProperty(s, key, toInteger(s[key]))\n",
    "YIELD node\n",
    "RETURN distinct 'done';\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uPqZhSJXSsIE"
   },
   "source": [
    "## Graph import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "0SgWfhB9SsIF"
   },
   "outputs": [],
   "source": [
    "with driver.session() as session:\n",
    "    for statement in import_queries.split(';'):\n",
    "        try:\n",
    "            session.run(statement.strip())\n",
    "        except:\n",
    "            pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pDc-7vY2SsIG"
   },
   "source": [
    "## Graph schema\n",
    "In the center of the graph, there are characters, also known as heroes. They can appear in multiple comics, are part of an event, and can belong to a group. For some of the characters, we also know their stats like speed and fighting skills. Finally, we have social ties between characters that represent relative, ally, or enemy relationships.\n",
    "\n",
    "There are 1105 characters that have appeared in 38875 comics.\n",
    "We have stats for 470 of the characters. There are also 92 groups and 74 events stored in the graph.\n",
    "## Exploratory graph analysis\n",
    "To get to know our graph, we will begin with a basic graph data exploration process. First, we will take a look at the characters that have most frequently appeared in comics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "5znOehe4SsIG",
    "outputId": "2637175d-e557-4577-8a63-d4fad1bac047"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>comics</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Spider-Man (1602)</td>\n",
       "      <td>3357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tony Stark</td>\n",
       "      <td>2354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Logan</td>\n",
       "      <td>2098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Steve Rogers</td>\n",
       "      <td>2019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thor (Marvel: Avengers Alliance)</td>\n",
       "      <td>1547</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          character  comics\n",
       "0                 Spider-Man (1602)    3357\n",
       "1                        Tony Stark    2354\n",
       "2                             Logan    2098\n",
       "3                      Steve Rogers    2019\n",
       "4  Thor (Marvel: Avengers Alliance)    1547"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)\n",
    "RETURN c.name as character, \n",
    "       count{ (c)-[:APPEARED_IN]->() } as comics\n",
    "ORDER BY comics DESC\n",
    "LIMIT 5\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DH8gYWHDSsII"
   },
   "source": [
    "The top five most frequent characters come as no surprise. Spiderman is the most frequent or popular character. It is no wonder that they created a younger version of Spiderman just recently, given his popularity. Tony Stark, also known as Iron Man, is in second place. It seems that Logan, also known as Wolverine, was quite popular throughout history, but I think that his popularity slowly faded away in recent times. Steve Rogers, who goes by the more popular name Captain America, is also quite famous. It would seem that the recent Marvel movies showcased the more popular characters from the comics.\n",
    "\n",
    "Next, we will look at how many comics were released throughout the decades. The year of the comic is stored as a string in our graph, so we can use the substringfunction to extract the decade."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "id": "WVFZR8EhSsIJ",
    "outputId": "6c89d8ca-c443-4e67-f451-5baa1d9017ed"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>decade</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1930</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1940</td>\n",
       "      <td>584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1950</td>\n",
       "      <td>756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1960</td>\n",
       "      <td>4114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1970</td>\n",
       "      <td>1956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1980</td>\n",
       "      <td>2428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1990</td>\n",
       "      <td>3738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2000</td>\n",
       "      <td>8309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2010</td>\n",
       "      <td>11139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2020</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>None</td>\n",
       "      <td>5737</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   decade  count\n",
       "0    1930     95\n",
       "1    1940    584\n",
       "2    1950    756\n",
       "3    1960   4114\n",
       "4    1970   1956\n",
       "5    1980   2428\n",
       "6    1990   3738\n",
       "7    2000   8309\n",
       "8    2010  11139\n",
       "9    2020     19\n",
       "10   None   5737"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Comic)\n",
    "RETURN substring(c.year, 0, 3) + \"0\" as decade, \n",
    "       count(*) as count\n",
    "ORDER BY decade ASC\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lDjnXxBYSsIJ"
   },
   "source": [
    "Interesting to see that the first comics were produced in the 1930s. Some of the heroes are relatively senior by now. There was a spike in the 1960s and then gradual progression over the decades with 11.139 comics in the 2010s. The last column represents comics with a null date, so for around 6000 comics out of 38000, we don’t have the date available. And we haven’t scraped all the comics in the 2020s either.\n",
    "\n",
    "Next, we will take a look at the most popular characters in the comics throughout the decades. We will iterate over comics and extract the top three most frequent heroes by the decade."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 332
    },
    "id": "r03jU04TSsIK",
    "outputId": "a6f64b13-a35c-4218-b653-792564b2c9c9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>decade</th>\n",
       "      <th>top_3_characters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1930</td>\n",
       "      <td>[Johnny Storm, Sub-Mariner, Archangel]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1940</td>\n",
       "      <td>[Johnny Storm, Two-Gun Kid, Steve Rogers]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1950</td>\n",
       "      <td>[Rawhide Kid, Tony Stark, Stephen Strange]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1960</td>\n",
       "      <td>[Thor (Marvel: Avengers Alliance), Spider-Man ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1970</td>\n",
       "      <td>[Spider-Man (1602), Stephen Strange, Shang-Chi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1980</td>\n",
       "      <td>[Logan, Tony Stark, Spider-Man (1602)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1990</td>\n",
       "      <td>[Spider-Man (1602), Tony Stark, Steve Rogers]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2000</td>\n",
       "      <td>[Spider-Man (1602), Tony Stark, Logan]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2010</td>\n",
       "      <td>[Spider-Man (1602), Steve Rogers, Logan]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  decade                                   top_3_characters\n",
       "0   1930             [Johnny Storm, Sub-Mariner, Archangel]\n",
       "1   1940          [Johnny Storm, Two-Gun Kid, Steve Rogers]\n",
       "2   1950         [Rawhide Kid, Tony Stark, Stephen Strange]\n",
       "3   1960  [Thor (Marvel: Avengers Alliance), Spider-Man ...\n",
       "4   1970  [Spider-Man (1602), Stephen Strange, Shang-Chi...\n",
       "5   1980             [Logan, Tony Stark, Spider-Man (1602)]\n",
       "6   1990      [Spider-Man (1602), Tony Stark, Steve Rogers]\n",
       "7   2000             [Spider-Man (1602), Tony Stark, Logan]\n",
       "8   2010           [Spider-Man (1602), Steve Rogers, Logan]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Comic)<-[:APPEARED_IN]-(c1:Character)\n",
    "WHERE NOT c.year = \"null\"\n",
    "WITH substring(c.year,0,3) + \"0\" as decade, \n",
    "     c1.name as character, \n",
    "     count(*) as count\n",
    "ORDER BY count DESC\n",
    "RETURN decade, collect(character)[..3] as top_3_characters\n",
    "ORDER BY decade\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K87WMRe8SsIK"
   },
   "source": [
    "It seems it all started with Johnny Storm, also known as the Human Torch. Iron Man (Tony Stark) was already popular in the 1950s, and Spiderman and Captain America (Steve Rogers) have risen in popularity in the 1960s. From then on, it seems that Spiderman, Wolverine, Iron Man, and Captain America win the popularity contest.\n",
    "You might be wondering what the events are in our graph, so let’s take a look. We will examine the events with the highest count of participating heroes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "-kYriTS7SsIL",
    "outputId": "b6240ecf-037e-4e21-ad4f-502ba24ae8ff"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>event</th>\n",
       "      <th>count_of_heroes</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fear Itself</td>\n",
       "      <td>132</td>\n",
       "      <td>2011-04-16 00:00:00</td>\n",
       "      <td>2011-10-18 00:00:00</td>\n",
       "      <td>The Serpent, God of Fear and brother to the Al...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dark Reign</td>\n",
       "      <td>128</td>\n",
       "      <td>2008-12-01 00:00:00</td>\n",
       "      <td>2009-12-31 12:59:00</td>\n",
       "      <td>Norman Osborn came out the hero of Secret Inva...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Acts of Vengeance!</td>\n",
       "      <td>93</td>\n",
       "      <td>1989-12-10 00:00:00</td>\n",
       "      <td>2008-01-04 00:00:00</td>\n",
       "      <td>Loki sets about convincing the super-villains ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Secret Invasion</td>\n",
       "      <td>89</td>\n",
       "      <td>2008-06-02 00:00:00</td>\n",
       "      <td>2009-01-25 00:00:00</td>\n",
       "      <td>The shape-shifting Skrulls have been infiltrat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Civil War</td>\n",
       "      <td>86</td>\n",
       "      <td>2006-07-01 00:00:00</td>\n",
       "      <td>2007-01-29 00:00:00</td>\n",
       "      <td>After a horrific tragedy raises questions on w...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                event  count_of_heroes                start  \\\n",
       "0         Fear Itself              132  2011-04-16 00:00:00   \n",
       "1          Dark Reign              128  2008-12-01 00:00:00   \n",
       "2  Acts of Vengeance!               93  1989-12-10 00:00:00   \n",
       "3     Secret Invasion               89  2008-06-02 00:00:00   \n",
       "4           Civil War               86  2006-07-01 00:00:00   \n",
       "\n",
       "                   end                                        description  \n",
       "0  2011-10-18 00:00:00  The Serpent, God of Fear and brother to the Al...  \n",
       "1  2009-12-31 12:59:00  Norman Osborn came out the hero of Secret Inva...  \n",
       "2  2008-01-04 00:00:00  Loki sets about convincing the super-villains ...  \n",
       "3  2009-01-25 00:00:00  The shape-shifting Skrulls have been infiltrat...  \n",
       "4  2007-01-29 00:00:00  After a horrific tragedy raises questions on w...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (e:Event)\n",
    "RETURN e.title as event, \n",
    "       count{ (e)<-[:PART_OF_EVENT]-() } as count_of_heroes,\n",
    "       e.start as start,\n",
    "       e.end as end,\n",
    "       e.description as description \n",
    "ORDER BY count_of_heroes DESC \n",
    "LIMIT 5\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nvfiqgprSsIL"
   },
   "source": [
    "I have little to no idea what these events represent, but it is interesting to see that many characters participate. Most of the events span over less than a year, while the Acts of Vengeance spans over two decades. And judging by the description, Loki had something to do with it along with 92! other characters. Unfortunately, we don’t have the connection between comics and events stored in our graph to allow further analysis. If someone will scrape the Marvel API, I will gladly add it to the dataset.\n",
    "\n",
    "Let’s also take a look at the biggest groups of characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "FiOynPnrSsIL",
    "outputId": "04a0dfc5-ed58-46ff-c296-104f0c580471"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>members</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>X-Men</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Avengers</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Defenders</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Next Avengers</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Guardians of the Galaxy</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     group  members\n",
       "0                    X-Men       41\n",
       "1                 Avengers       31\n",
       "2                Defenders       26\n",
       "3            Next Avengers       14\n",
       "4  Guardians of the Galaxy       12"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (g:Group)\n",
    "RETURN g.name as group, \n",
    "       count{ (g)<-[:PART_OF_GROUP]-() } as members\n",
    "ORDER BY members DESC LIMIT 5\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W-oBTLSySsIM"
   },
   "source": [
    "There are 41 characters in X-Men, which makes sense as they had a whole academy. You might be surprised by 31 members of Avengers, but in the comics, there were many members of Avengers, although most are former members.\n",
    "\n",
    "Just because we can, let’s inspect if some members of the same group are also enemies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "4_jPXMWOSsIM",
    "outputId": "36d2343b-df69-449a-e0f8-01c97b1fb7e0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>character1</th>\n",
       "      <th>character2</th>\n",
       "      <th>group</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logan</td>\n",
       "      <td>Sabretooth (House of M)</td>\n",
       "      <td>X-Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Logan</td>\n",
       "      <td>Mystique (House of M)</td>\n",
       "      <td>X-Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CAIN MARKO JUGGERNAUT</td>\n",
       "      <td>Logan</td>\n",
       "      <td>X-Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CAIN MARKO JUGGERNAUT</td>\n",
       "      <td>Storm (Marvel Heroes)</td>\n",
       "      <td>X-Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Rogue (X-Men: Battle of the Atom)</td>\n",
       "      <td>Warren Worthington III</td>\n",
       "      <td>X-Men</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          character1               character2  group\n",
       "0                              Logan  Sabretooth (House of M)  X-Men\n",
       "1                              Logan    Mystique (House of M)  X-Men\n",
       "2              CAIN MARKO JUGGERNAUT                    Logan  X-Men\n",
       "3              CAIN MARKO JUGGERNAUT    Storm (Marvel Heroes)  X-Men\n",
       "4  Rogue (X-Men: Battle of the Atom)   Warren Worthington III  X-Men"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c1:Character)-[:PART_OF_GROUP]->(g:Group)<-[:PART_OF_GROUP]-(c2:Character)\n",
    "WHERE (c1)-[:ENEMY]-(c2) and id(c1) < id(c2)\n",
    "RETURN c1.name as character1, c2.name as character2, g.name as group\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7a97nGmbSsIN"
   },
   "source": [
    "It seems that Logan does not get along with some of the other X-Men. For some of the characters, we also have the place of origin and education available, so let’s quickly look at that. During the scraping, I noticed a hero originated from Yugoslavia, so I wonder if there are more characters from Yugoslavia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "id": "VyWjGhDmSsIN",
    "outputId": "01f06941-5ac1-4c74-a7ad-fc8427157e79"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>place_of_origin</th>\n",
       "      <th>aliases</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Purple Man</td>\n",
       "      <td>Rijeka, Yugoslavia</td>\n",
       "      <td>Killgrave the Purple Man, Killy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Abomination (Ultimate)</td>\n",
       "      <td>Zagreb, Yugoslavia</td>\n",
       "      <td>Agent R-7, the Ravager of Worlds</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                character     place_of_origin  \\\n",
       "0              Purple Man  Rijeka, Yugoslavia   \n",
       "1  Abomination (Ultimate)  Zagreb, Yugoslavia   \n",
       "\n",
       "                            aliases  \n",
       "0   Killgrave the Purple Man, Killy  \n",
       "1  Agent R-7, the Ravager of Worlds  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)\n",
    "WHERE c.place_of_origin contains \"Yugoslavia\"\n",
    "RETURN c.name as character, \n",
    "       c.place_of_origin as place_of_origin,\n",
    "       c.aliases as aliases\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hOqK29i2SsIN"
   },
   "source": [
    "Two characters originated from today’s Croatia, which is less than two hours drive from where I live. Let’s also check out all the characters that completed their Ph.D. degree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "jkBUu0vqSsIO",
    "outputId": "92e6148e-dca1-492c-9256-b7687bfe3677"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>education</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Radioactive Man</td>\n",
       "      <td>Ph.D. in physics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Killmonger</td>\n",
       "      <td>Ph.D. in engineering and MBA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nightshade</td>\n",
       "      <td>Extensively self-taught in multiple discipline...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Humbug</td>\n",
       "      <td>Ph.D. in entomology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>UNKNOWN ACHEBE</td>\n",
       "      <td>Ph.D. in Law (Yale), degrees in Psychology, Po...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>PROFESSOR MENDEL STROMM MENDEL STROMM</td>\n",
       "      <td>Ph.D. in robotics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>FRANKLIN HALL GRAVITON</td>\n",
       "      <td>Ph.D. in physics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Morbius</td>\n",
       "      <td>Ph.D in Biochemistry</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Tony Stark</td>\n",
       "      <td>Ph.Ds in physics and electrical engineering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Hulk-dok</td>\n",
       "      <td>Ph.D in nuclear physics and two other fields</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               character  \\\n",
       "0                        Radioactive Man   \n",
       "1                             Killmonger   \n",
       "2                             Nightshade   \n",
       "3                                 Humbug   \n",
       "4                         UNKNOWN ACHEBE   \n",
       "5  PROFESSOR MENDEL STROMM MENDEL STROMM   \n",
       "6                 FRANKLIN HALL GRAVITON   \n",
       "7                                Morbius   \n",
       "8                             Tony Stark   \n",
       "9                               Hulk-dok   \n",
       "\n",
       "                                           education  \n",
       "0                                   Ph.D. in physics  \n",
       "1                       Ph.D. in engineering and MBA  \n",
       "2  Extensively self-taught in multiple discipline...  \n",
       "3                                Ph.D. in entomology  \n",
       "4  Ph.D. in Law (Yale), degrees in Psychology, Po...  \n",
       "5                                  Ph.D. in robotics  \n",
       "6                                   Ph.D. in physics  \n",
       "7                               Ph.D in Biochemistry  \n",
       "8        Ph.Ds in physics and electrical engineering  \n",
       "9       Ph.D in nuclear physics and two other fields  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)\n",
    "WHERE c.education contains \"Ph.D\"\n",
    "RETURN c.name as character, c.education as education\n",
    "LIMIT 10\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64g3z5ZKSsIO"
   },
   "source": [
    "It looks like a lot of these heroes are quite employable. Only Nightshade seems a bit dodgy. It feels like something one would put on their LinkedIn profile to get noticed when searching for Ph.D. profiles. By the way, did you know that Professor X has four Ph.D.s and is also MD in psychiatry? Quite the educated men.\n",
    "## Analyzing communities of allies and relatives\n",
    "We have examined basic graph statistics, and now we will focus more on network analysis. We will investigate the social ties between characters.\n",
    "To start, we will calculate the degree values for each relationship type between characters and display the heroes with the highest overall degree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "ngpqol3mSsIO",
    "outputId": "01481c9d-4f13-4b4f-ac79-0a643b3ae2ea"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>allies</th>\n",
       "      <th>enemies</th>\n",
       "      <th>relative</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Scarlet Witch (Marvel Heroes)</td>\n",
       "      <td>16</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Thor (Marvel: Avengers Alliance)</td>\n",
       "      <td>9</td>\n",
       "      <td>14</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Invisible Woman (Marvel: Avengers Alliance)</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Logan</td>\n",
       "      <td>14</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Karnak</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          name  allies  enemies  relative\n",
       "0                Scarlet Witch (Marvel Heroes)      16       14         8\n",
       "1             Thor (Marvel: Avengers Alliance)       9       14        10\n",
       "2  Invisible Woman (Marvel: Avengers Alliance)      13       10         7\n",
       "3                                        Logan      14       10         5\n",
       "4                                       Karnak       6        2        17"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)\n",
    "RETURN c.name as name,\n",
    "       count{ (c)-[:ALLY]->() } as allies,\n",
    "       count{ (c)-[:ENEMY]->() } as enemies,\n",
    "       count{ (c)-[:RELATIVE]->() } as relative\n",
    "ORDER BY allies + enemies + relative DESC \n",
    "LIMIT 5\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e8zoVxfvSsIP"
   },
   "source": [
    "Scarlet Witch and Thor seem to have the most direct enemies. Wolverine has the most allies but also many enemies. It looks like Triton has a big family with 17 direct relative relationships. We can use the `apoc.path.subgraphAll` procedure to examine the relatives' community of Triton."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "iuYqg9DNSsIP",
    "outputId": "34bd025f-efd6-4005-f011-9e911f08c1f8"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodes</th>\n",
       "      <th>relationships</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[(aliases, education, identity, name, id, plac...</td>\n",
       "      <td>[(), (), (), (), (), (), (), (), (), (), (), (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               nodes  \\\n",
       "0  [(aliases, education, identity, name, id, plac...   \n",
       "\n",
       "                                       relationships  \n",
       "0  [(), (), (), (), (), (), (), (), (), (), (), (...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH p=(c:Character{name:\"Triton\"})\n",
    "CALL apoc.path.subgraphAll(id(c), {relationshipFilter:\"RELATIVE\"})\n",
    "YIELD nodes, relationships\n",
    "RETURN nodes, relationships\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ULqVl7c1SsIP"
   },
   "source": [
    "I never knew that some of the Marvel heroes have quite a big happy family. It wouldn’t be accurate if there weren’t a black sheep of the family present. Maximus looks like the family’s black sheep here as he has four enemies within the family. You might wonder why ally and enemy relationships are shown when we only traversed the relative ties. Neo4j Browser has a feature that displays all connections between nodes on the screen.\n",
    "\n",
    "## Weakly Connected Components algorithm\n",
    "The Weakly Connected Components is a part of almost every graph analysis workflow. It is used to find disconnected components or islands within the network. In this example, the graph consists of two components. Michael, Mark, and Doug belong to the first component, while Bridget, Alice, and Charles belong to the second component. We will apply the Weakly Connected Components algorithm to find the largest component of allied characters. As we don’t plan to run any other algorithms on this network, we will use the anonymous graph projection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "bRJ6FcK3thiW",
    "outputId": "e2f25953-af46-4dc6-d348-388253b1731c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodeProjection</th>\n",
       "      <th>relationshipProjection</th>\n",
       "      <th>graphName</th>\n",
       "      <th>nodeCount</th>\n",
       "      <th>relationshipCount</th>\n",
       "      <th>projectMillis</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'Character': {'label': 'Character', 'properti...</td>\n",
       "      <td>{'ALLY': {'orientation': 'NATURAL', 'indexInve...</td>\n",
       "      <td>allies</td>\n",
       "      <td>1105</td>\n",
       "      <td>313</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      nodeProjection  \\\n",
       "0  {'Character': {'label': 'Character', 'properti...   \n",
       "\n",
       "                              relationshipProjection graphName  nodeCount  \\\n",
       "0  {'ALLY': {'orientation': 'NATURAL', 'indexInve...    allies       1105   \n",
       "\n",
       "   relationshipCount  projectMillis  \n",
       "0                313             32  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.graph.project('allies', 'Character', 'ALLY')\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "_sTRSNI4SsIP",
    "outputId": "cd39e100-0d26-41ad-f92a-74eb034da933"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>componentId</th>\n",
       "      <th>members</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>152</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>371</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>132</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>128</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   componentId  members\n",
       "0            0      195\n",
       "1          152        4\n",
       "2          371        3\n",
       "3          132        2\n",
       "4          128        2"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.wcc.stream('allies')\n",
    "YIELD nodeId, componentId\n",
    "WITH componentId, count(*) as members\n",
    "WHERE members > 1\n",
    "RETURN componentId, members\n",
    "ORDER BY members DESC\n",
    "LIMIT 5\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GIoYK30eSsIQ"
   },
   "source": [
    "The largest component of allies has 195 members. Then we have a couple of tiny allies islands with only a few members. If we visualize the largest component of allies in the Neo4j Browser and have the connect results nodes option selected, we get the following visualization.\n",
    "\n",
    "Although we have found the largest allies component, we can observe that many of the characters in the component are actually enemies (red relationships). To better understand why this occurs, let’s look at the following example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xRX4udRpSsIQ"
   },
   "source": [
    "## Custom ally component algorithm\n",
    "Suppose we wanted to find communities of allies where there are no enemies within the given component. The algorithm implementation is relatively straightforward, and you could use Neo4j custom procedures, for example. Still, if you are like me and don’t speak Java, you can always resort to your favorite scripting language. I have developed the custom Ally component algorithm in Python. First, we define some helper functions for fetching allies and enemies of a single node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "Mws_aYp3SsIQ"
   },
   "outputs": [],
   "source": [
    "def get_allies(node_id):\n",
    "    data = session.run(\"\"\"MATCH (c:Character)-[:ALLY]-(ally) \n",
    "                          WHERE c.id = $node_id \n",
    "                          RETURN collect(ally.id) as allies\"\"\",\n",
    "                      {'node_id':node_id})\n",
    "    return data.single()['allies']\n",
    "\n",
    "def get_enemies(node_id):\n",
    "    data = session.run(\"\"\"MATCH (c:Character)-[:ENEMY]-(enemy) \n",
    "                          WHERE c.id = $node_id \n",
    "                          RETURN collect(enemy.id) as allies\"\"\",\n",
    "                      {'node_id':node_id})\n",
    "    return data.single()['allies']\n",
    "\n",
    "def get_members():\n",
    "    return session.run(\"\"\"\n",
    "    CALL gds.wcc.stream('allies')\n",
    "    YIELD nodeId, componentId\n",
    "    WITH componentId, collect(gds.util.asNode(nodeId).id) as members\n",
    "    WHERE size(members) > 10\n",
    "    RETURN componentId, members\n",
    "    \"\"\").single()['members']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c0gNv0AvSsIR"
   },
   "source": [
    "My implementation is relatively simple. The input to the algorithm is the list of all node ids in the largest allied components. Start from a single node, load its enemies into the enemies list and load its allies into a queue that will be processed later. Then we iterate over the allied queue. If a node is not an enemy with any of the existing nodes in the component, add them to the community list and add their enemies to the community’s enemies list. I’ve added some minor performance tweaks like if we have traversed the node already in the allies queue, we can remove that node from the global list of starting nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "9VelqZB5SsIR"
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "def get_largest_stable_allies(node_list):\n",
    "    final_communities = list()\n",
    "    while node_list:\n",
    "        community = set()\n",
    "        enemies_list = set()\n",
    "        visited = set()\n",
    "        \n",
    "        allies_list = deque()\n",
    "        allies_list.appendleft(node_list[0])\n",
    "        \n",
    "        while allies_list:\n",
    "            # Get the node from the queue\n",
    "            start_node = allies_list.pop()\n",
    "            \n",
    "            # Skip if current node is enemy with anyone\n",
    "            if start_node in enemies_list:\n",
    "                continue\n",
    "            \n",
    "            # Get allies and enemies\n",
    "            allies = get_allies(start_node)\n",
    "            enemies = get_enemies(start_node)\n",
    "            \n",
    "            visited.add(start_node)\n",
    "            # Add enemies\n",
    "            enemies_list.update(enemies)\n",
    "            # Add allies to the list of next visits\n",
    "            allies_list.extendleft([x for x in allies if (x not in enemies_list) and (x not in visited)])\n",
    "            # Add current node to community\n",
    "            community.add(start_node)\n",
    "            \n",
    "            # Remove visited nodes from global node list\n",
    "            try:\n",
    "                node_list.remove(start_node)\n",
    "            except:\n",
    "                pass\n",
    "        final_communities.append(list(community))\n",
    "    return max(final_communities, key=len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2h03aJT2SsIR",
    "outputId": "a81deb75-7b02-4eac-dedc-0ced7f93c7e8"
   },
   "outputs": [],
   "source": [
    "members = get_members()\n",
    "get_largest_stable_allies(members)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q-8YOnneSsIS"
   },
   "source": [
    "In this code, the algorithm only returns the ids of nodes that belong to the largest allied component where there are no enemies within. It shouldn’t be a problem to mark these nodes in Neo4j, as you can match them by their ids. The largest component of allies, where there are no enemies within, has 142 members. If we visualize it in Neo4j Browser, we can see that there are no enemy relationships visible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "sJjJmZLju4a2",
    "outputId": "eaaf61f0-6d12-43d4-9dc9-a92defb21900"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>graphName</th>\n",
       "      <th>database</th>\n",
       "      <th>memoryUsage</th>\n",
       "      <th>sizeInBytes</th>\n",
       "      <th>nodeCount</th>\n",
       "      <th>relationshipCount</th>\n",
       "      <th>configuration</th>\n",
       "      <th>density</th>\n",
       "      <th>creationTime</th>\n",
       "      <th>modificationTime</th>\n",
       "      <th>schema</th>\n",
       "      <th>schemaWithOrientation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>allies</td>\n",
       "      <td>neo4j</td>\n",
       "      <td></td>\n",
       "      <td>-1</td>\n",
       "      <td>1105</td>\n",
       "      <td>313</td>\n",
       "      <td>{'relationshipProjection': {'ALLY': {'orientat...</td>\n",
       "      <td>0.000257</td>\n",
       "      <td>2023-02-01T15:30:42.693998218+00:00</td>\n",
       "      <td>2023-02-01T15:30:42.725535343+00:00</td>\n",
       "      <td>{'graphProperties': {}, 'relationships': {'ALL...</td>\n",
       "      <td>{'graphProperties': {}, 'relationships': {'ALL...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  graphName database memoryUsage  sizeInBytes  nodeCount  relationshipCount  \\\n",
       "0    allies    neo4j                       -1       1105                313   \n",
       "\n",
       "                                       configuration   density  \\\n",
       "0  {'relationshipProjection': {'ALLY': {'orientat...  0.000257   \n",
       "\n",
       "                          creationTime                     modificationTime  \\\n",
       "0  2023-02-01T15:30:42.693998218+00:00  2023-02-01T15:30:42.725535343+00:00   \n",
       "\n",
       "                                              schema  \\\n",
       "0  {'graphProperties': {}, 'relationships': {'ALL...   \n",
       "\n",
       "                               schemaWithOrientation  \n",
       "0  {'graphProperties': {}, 'relationships': {'ALL...  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.graph.drop('allies')\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c0zjM88gSsIS"
   },
   "source": [
    "## Analyzing characters’ stats\n",
    "In the last part of our analysis, we will examine the stats of the characters. We have the stats available for a total of 470 heroes. This information was scraped from Marvel’s website. The scale for stats ranges from zero to seven, and Iron Man does not have a single seven. Probably not the strongest of the heroes, even though he is one of the more popular ones. Now we will explore the characters with the highest stats average. Whenever I need some help with my cypher queries, I turn to Neo4j Slack. Luckily for us, Andrew Bowman is always around with great advice on optimizing and prettifying our cypher queries. This time he showed me the `apoc.map.values` procedure. It can be used to fetch all properties of a single node without explicitly writing the property keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "Hq_NzZ4qSsIS",
    "outputId": "954cbafd-6973-4256-fdf1-bcf2fb837fd6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>character</th>\n",
       "      <th>average_stats</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Sasquatch (Walter Langkowski)</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Squirrel Girl</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Galactus</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Deathstrike (Ultimate)</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GRAYDON CREED</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>CHTHON</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>UNREVEALED; GAEA IS HER GREEK NAME GAEA</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>SET</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Rogue (X-Men: Battle of the Atom)</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Legion</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 character  average_stats\n",
       "0            Sasquatch (Walter Langkowski)            7.0\n",
       "1                            Squirrel Girl            7.0\n",
       "2                                 Galactus            7.0\n",
       "3                   Deathstrike (Ultimate)            7.0\n",
       "4                            GRAYDON CREED            7.0\n",
       "5                                   CHTHON            7.0\n",
       "6  UNREVEALED; GAEA IS HER GREEK NAME GAEA            7.0\n",
       "7                                      SET            7.0\n",
       "8        Rogue (X-Men: Battle of the Atom)            7.0\n",
       "9                                   Legion            7.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)-[:HAS_STATS]->(stats)\n",
    "RETURN c.name as character, \n",
    "       apoc.coll.avg(apoc.map.values(stats, keys(stats))) as average_stats\n",
    "ORDER BY average_stats DESC\n",
    "LIMIT 10\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S6eQ0u4ESsIT"
   },
   "source": [
    "It seems many characters have their stats maxed out. I am not sure exactly how this data collection process works, but I found a fascinating heroine by the name of Squirrel Girl that could probably kick Iron Man’s ass with one hand while making sourdough bread with the other. Or polish her nails, not exactly sure what type of girl she is. The only thing certain is that she is a badass.\n",
    "## k-Nearest Neighbours algorithm\n",
    "The k-Nearest Neighbour is one of the more standard graph algorithms and was already implemented in the Graph Data Science library before in the form of Cosine, Euclidian, and Pearson similarity algorithms. Those were basic implementation where the algorithms compared a given vector for all node pairs in the network. Because comparing all node pairs does not scale well, another implementation of the kNN algorithm was added to the library. It is based on the Efficient k-nearest neighbor graph construction for generic similarity measures article. Instead of comparing every node pair, the algorithm selects possible neighbors based on the assumption that the neighbors-of-neighbors of a node are most likely already the nearest one. The algorithm scales quasi-linear with respect to the node count instead of being quadratic. The implementation uses the Cosine similarity to compare two vectors.\n",
    "First, we need to create a vector (array of numbers) that will be compared between the pairs of heroes. We will use the characters’ stats as well as their ability to fly to populate the vector. Because all stats have the same range between zero and seven, there is no need for normalization. We only need to encode the flight feature to span between zero and seven as well. Those characters that can fly will have the value of flight feature seven, while those who can’t fly will have the value zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "HnARtBNHSsIT",
    "outputId": "36ed9ea8-668b-4e1f-be5c-11aac8ceddad"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)-[:HAS_STATS]->(s)\n",
    "WITH c, [s.durability, s.energy, s.fighting_skills, \n",
    "         s.intelligence, s.speed, s.strength,\n",
    "         CASE WHEN c.flight = 'true' THEN 7 ELSE 0 END] as stats_vector\n",
    "SET c.stats_vector = stats_vector\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jBJcE8-tSsIT"
   },
   "source": [
    "We will also tag the characters that have the stats vector with a second label. This way, we can easily filter heroes with a stats vector in our native projection of the named graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "C9FhxG_uSsIT",
    "outputId": "304c8dd9-680c-4bec-eca1-068862baf3c5"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)\n",
    "WHERE c.stats_vector IS NOT NULL\n",
    "SET c:CharacterStats\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VlIOJaADSsIT"
   },
   "source": [
    "Now that everything is ready, we can go ahead and load our named graph. We will project all nodes with the CharacterStats label and their stats_vector properties in a named graph. If you need a quick refresher or introduction to how the GDS library works, I would suggest taking the Introduction to Graph Algorithms course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "YBv0q7zESsIU",
    "outputId": "dc5281e8-960f-4090-adb6-4eb597af3e6d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nodeProjection</th>\n",
       "      <th>relationshipProjection</th>\n",
       "      <th>graphName</th>\n",
       "      <th>nodeCount</th>\n",
       "      <th>relationshipCount</th>\n",
       "      <th>projectMillis</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'CharacterStats': {'label': 'CharacterStats',...</td>\n",
       "      <td>{'__ALL__': {'orientation': 'NATURAL', 'indexI...</td>\n",
       "      <td>marvel</td>\n",
       "      <td>470</td>\n",
       "      <td>515</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      nodeProjection  \\\n",
       "0  {'CharacterStats': {'label': 'CharacterStats',...   \n",
       "\n",
       "                              relationshipProjection graphName  nodeCount  \\\n",
       "0  {'__ALL__': {'orientation': 'NATURAL', 'indexI...    marvel        470   \n",
       "\n",
       "   relationshipCount  projectMillis  \n",
       "0                515             38  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.graph.project('marvel', 'CharacterStats',\n",
    "  '*', {nodeProperties:'stats_vector'})\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zqmGpbBfSsIU"
   },
   "source": [
    "Now, we can go ahead and infer the similarity network with the new kNN algorithm. We will use the mutate mode of the algorithm. The mutate mode stores the results back to the projected graph instead of the Neo4j stored graph. This way, we can use the kNN algorithm results as the input for the community detection algorithms later in the workflow. The kNN algorithm has some parameters we can use to fine-tune the results:\n",
    "* topK: The number of neighbors to find for each node. The K-nearest neighbors are returned.\n",
    "* sampleRate: Sample rate to limit the number of comparisons per node.\n",
    "* deltaThreshold: Value as a percentage to determine when to stop early. If fewer updates than the configured value happen, the algorithm stops.\n",
    "* randomJoins: Between every iteration, how many attempts are being made to connect new node neighbors based on random selection.\n",
    "\n",
    "We will define the topK value of 15 and sampleRate of 0.8, and leave the other parameters at default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 159
    },
    "id": "2ZxBxqg4SsIU",
    "outputId": "809af1c6-e74a-4f45-e8c5-8b5f9d74f5c0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ranIterations</th>\n",
       "      <th>nodePairsConsidered</th>\n",
       "      <th>didConverge</th>\n",
       "      <th>preProcessingMillis</th>\n",
       "      <th>computeMillis</th>\n",
       "      <th>mutateMillis</th>\n",
       "      <th>postProcessingMillis</th>\n",
       "      <th>nodesCompared</th>\n",
       "      <th>relationshipsWritten</th>\n",
       "      <th>similarityDistribution</th>\n",
       "      <th>configuration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>377254</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>99</td>\n",
       "      <td>22</td>\n",
       "      <td>-1</td>\n",
       "      <td>470</td>\n",
       "      <td>7050</td>\n",
       "      <td>{'p1': 0.40000057220458984, 'max': 1.000006675...</td>\n",
       "      <td>{'topK': 15, 'maxIterations': 100, 'randomJoin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ranIterations  nodePairsConsidered  didConverge  preProcessingMillis  \\\n",
       "0              4               377254         True                    0   \n",
       "\n",
       "   computeMillis  mutateMillis  postProcessingMillis  nodesCompared  \\\n",
       "0             99            22                    -1            470   \n",
       "\n",
       "   relationshipsWritten                             similarityDistribution  \\\n",
       "0                  7050  {'p1': 0.40000057220458984, 'max': 1.000006675...   \n",
       "\n",
       "                                       configuration  \n",
       "0  {'topK': 15, 'maxIterations': 100, 'randomJoin...  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.knn.mutate('marvel', {nodeProperties:'stats_vector', \n",
    "  sampleRate:0.8, topK:15, mutateProperty:'score', mutateRelationshipType:'SIMILAR'})\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JgZtHETpSsIU"
   },
   "source": [
    "## Louvain Modularity algorithm\n",
    "The similarity network is inferred and stored in the named graph. We can examine the community structure of this new similarity network with the Louvain Modularity algorithm. As the similarity scores of relationships are available as their properties, we will use the weighted variant of the Louvain Modularity algorithm. Using the `relationshipWeightProperty` parameter, we let the algorithm know it should consider the relationships’ weight when calculating the network’s community structure. This time we will use the `write` mode of the algorithm to store the results back to the Neo4j stored graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "hvFThW3mSsIU",
    "outputId": "401bc3ad-2393-4047-9f8e-a2150160a1ef"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>writeMillis</th>\n",
       "      <th>nodePropertiesWritten</th>\n",
       "      <th>modularity</th>\n",
       "      <th>modularities</th>\n",
       "      <th>ranLevels</th>\n",
       "      <th>communityCount</th>\n",
       "      <th>communityDistribution</th>\n",
       "      <th>postProcessingMillis</th>\n",
       "      <th>preProcessingMillis</th>\n",
       "      <th>computeMillis</th>\n",
       "      <th>configuration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>24</td>\n",
       "      <td>470</td>\n",
       "      <td>0.620696</td>\n",
       "      <td>[0.5416420627544132, 0.6206963277517245]</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>{'p99': 115, 'min': 10, 'max': 115, 'mean': 58...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1900</td>\n",
       "      <td>{'maxIterations': 10, 'writeConcurrency': 4, '...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   writeMillis  nodePropertiesWritten  modularity  \\\n",
       "0           24                    470    0.620696   \n",
       "\n",
       "                               modularities  ranLevels  communityCount  \\\n",
       "0  [0.5416420627544132, 0.6206963277517245]          2               8   \n",
       "\n",
       "                               communityDistribution  postProcessingMillis  \\\n",
       "0  {'p99': 115, 'min': 10, 'max': 115, 'mean': 58...                     3   \n",
       "\n",
       "   preProcessingMillis  computeMillis  \\\n",
       "0                    0           1900   \n",
       "\n",
       "                                       configuration  \n",
       "0  {'maxIterations': 10, 'writeConcurrency': 4, '...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.louvain.write('marvel',\n",
    "  {relationshipTypes:['SIMILAR'],  \n",
    "   relationshipWeightProperty:'score', \n",
    "   writeProperty:'louvain'});\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nw6aA0KWSsIV"
   },
   "source": [
    "We can examine the community structure results with the following cypher query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "iri4abwxSsIV",
    "outputId": "41300bee-b113-479d-863c-8a42349a8caa"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>community</th>\n",
       "      <th>members</th>\n",
       "      <th>fighting_skills</th>\n",
       "      <th>durability</th>\n",
       "      <th>energy</th>\n",
       "      <th>intelligence</th>\n",
       "      <th>speed</th>\n",
       "      <th>strength</th>\n",
       "      <th>flight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>166</td>\n",
       "      <td>76</td>\n",
       "      <td>4.460526</td>\n",
       "      <td>4.723684</td>\n",
       "      <td>3.960526</td>\n",
       "      <td>3.894737</td>\n",
       "      <td>4.105263</td>\n",
       "      <td>4.381579</td>\n",
       "      <td>0.460526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>322</td>\n",
       "      <td>115</td>\n",
       "      <td>3.052174</td>\n",
       "      <td>2.530435</td>\n",
       "      <td>1.973913</td>\n",
       "      <td>2.921739</td>\n",
       "      <td>2.365217</td>\n",
       "      <td>2.269565</td>\n",
       "      <td>0.547826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>178</td>\n",
       "      <td>61</td>\n",
       "      <td>4.147541</td>\n",
       "      <td>3.426230</td>\n",
       "      <td>2.196721</td>\n",
       "      <td>3.098361</td>\n",
       "      <td>2.950820</td>\n",
       "      <td>3.295082</td>\n",
       "      <td>0.229508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>235</td>\n",
       "      <td>29</td>\n",
       "      <td>4.827586</td>\n",
       "      <td>5.379310</td>\n",
       "      <td>4.310345</td>\n",
       "      <td>4.551724</td>\n",
       "      <td>4.206897</td>\n",
       "      <td>4.931034</td>\n",
       "      <td>0.241379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>358</td>\n",
       "      <td>10</td>\n",
       "      <td>1.900000</td>\n",
       "      <td>0.900000</td>\n",
       "      <td>1.300000</td>\n",
       "      <td>3.300000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>351</td>\n",
       "      <td>85</td>\n",
       "      <td>3.611765</td>\n",
       "      <td>4.200000</td>\n",
       "      <td>2.635294</td>\n",
       "      <td>3.200000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.741176</td>\n",
       "      <td>1.070588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>61</td>\n",
       "      <td>47</td>\n",
       "      <td>5.829787</td>\n",
       "      <td>6.744681</td>\n",
       "      <td>6.617021</td>\n",
       "      <td>6.276596</td>\n",
       "      <td>6.787234</td>\n",
       "      <td>6.574468</td>\n",
       "      <td>2.085106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>220</td>\n",
       "      <td>47</td>\n",
       "      <td>4.489362</td>\n",
       "      <td>6.042553</td>\n",
       "      <td>5.553191</td>\n",
       "      <td>4.234043</td>\n",
       "      <td>5.638298</td>\n",
       "      <td>5.446809</td>\n",
       "      <td>2.680851</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   community  members  fighting_skills  durability    energy  intelligence  \\\n",
       "0        166       76         4.460526    4.723684  3.960526      3.894737   \n",
       "1        322      115         3.052174    2.530435  1.973913      2.921739   \n",
       "2        178       61         4.147541    3.426230  2.196721      3.098361   \n",
       "3        235       29         4.827586    5.379310  4.310345      4.551724   \n",
       "4        358       10         1.900000    0.900000  1.300000      3.300000   \n",
       "5        351       85         3.611765    4.200000  2.635294      3.200000   \n",
       "6         61       47         5.829787    6.744681  6.617021      6.276596   \n",
       "7        220       47         4.489362    6.042553  5.553191      4.234043   \n",
       "\n",
       "      speed  strength    flight  \n",
       "0  4.105263  4.381579  0.460526  \n",
       "1  2.365217  2.269565  0.547826  \n",
       "2  2.950820  3.295082  0.229508  \n",
       "3  4.206897  4.931034  0.241379  \n",
       "4  0.800000  0.700000  0.000000  \n",
       "5  3.000000  3.741176  1.070588  \n",
       "6  6.787234  6.574468  2.085106  \n",
       "7  5.638298  5.446809  2.680851  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)-[:HAS_STATS]->(stats)\n",
    "RETURN c.louvain as community, count(*) as members, \n",
    "       avg(stats.fighting_skills) as fighting_skills,\n",
    "       avg(stats.durability) as durability,\n",
    "       avg(stats.energy) as energy,\n",
    "       avg(stats.intelligence) as intelligence,\n",
    "       avg(stats.speed) as speed,\n",
    "       avg(stats.strength) as strength,\n",
    "       avg(CASE WHEN c.flight = 'true' THEN 7.0 ELSE 0.0 END) as flight\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EuDqexP_SsIV"
   },
   "source": [
    "It would make sense to add the standard deviation for each stat, but it wouldn’t be presentable for a blog post. The community with an id 27 has the most powerful members. The average for most stats is 6.5, which means that they are almost entirely maxed out. The average value of flight at 2 indicates that around 30% (2/7) of the members can fly. The largest community with 106 members has their stats averaged between 2 and 3, which would indicate that they might be support characters with lesser abilities. The characters with stronger abilities are usually the lead characters.\n",
    "\n",
    "## Label Propagation algorithm\n",
    "Label Propagation algorithm can also be used to determine the community structure of a network. We will apply it to the inferred similarity network and compare the results with the Louvain Modularity algorithm results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 142
    },
    "id": "Swa-elcbSsIV",
    "outputId": "d1a8bc89-c333-496d-95b4-4c4e6014bdae"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>writeMillis</th>\n",
       "      <th>nodePropertiesWritten</th>\n",
       "      <th>ranIterations</th>\n",
       "      <th>didConverge</th>\n",
       "      <th>communityCount</th>\n",
       "      <th>communityDistribution</th>\n",
       "      <th>postProcessingMillis</th>\n",
       "      <th>preProcessingMillis</th>\n",
       "      <th>computeMillis</th>\n",
       "      <th>configuration</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>113</td>\n",
       "      <td>470</td>\n",
       "      <td>10</td>\n",
       "      <td>False</td>\n",
       "      <td>9</td>\n",
       "      <td>{'p99': 173, 'min': 6, 'max': 173, 'mean': 52....</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>45</td>\n",
       "      <td>{'jobId': '63ffc66d-5123-44ef-a8ee-edd7dd60f95...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   writeMillis  nodePropertiesWritten  ranIterations  didConverge  \\\n",
       "0          113                    470             10        False   \n",
       "\n",
       "   communityCount                              communityDistribution  \\\n",
       "0               9  {'p99': 173, 'min': 6, 'max': 173, 'mean': 52....   \n",
       "\n",
       "   postProcessingMillis  preProcessingMillis  computeMillis  \\\n",
       "0                     3                    0             45   \n",
       "\n",
       "                                       configuration  \n",
       "0  {'jobId': '63ffc66d-5123-44ef-a8ee-edd7dd60f95...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "CALL gds.labelPropagation.write('marvel',\n",
    "  {relationshipTypes:['SIMILAR'],\n",
    "   relationshipWeightProperty:'score', \n",
    "   writeProperty:'labelPropagation'})\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o-XFMfXYSsIV"
   },
   "source": [
    "We investigate the results of the Label Propagation algorithm.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 457
    },
    "id": "x3bMD8sySsIW",
    "outputId": "7617bf5f-8262-4a16-e9df-9503067867ff"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>community</th>\n",
       "      <th>members</th>\n",
       "      <th>fighting_skills</th>\n",
       "      <th>durability</th>\n",
       "      <th>energy</th>\n",
       "      <th>intelligence</th>\n",
       "      <th>speed</th>\n",
       "      <th>strength</th>\n",
       "      <th>flight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>570</td>\n",
       "      <td>10</td>\n",
       "      <td>4.300000</td>\n",
       "      <td>5.600000</td>\n",
       "      <td>4.800000</td>\n",
       "      <td>4.400000</td>\n",
       "      <td>5.100000</td>\n",
       "      <td>4.700000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>451</td>\n",
       "      <td>173</td>\n",
       "      <td>3.225434</td>\n",
       "      <td>2.815029</td>\n",
       "      <td>2.075145</td>\n",
       "      <td>3.005780</td>\n",
       "      <td>2.358382</td>\n",
       "      <td>2.502890</td>\n",
       "      <td>0.566474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>356</td>\n",
       "      <td>58</td>\n",
       "      <td>4.551724</td>\n",
       "      <td>4.879310</td>\n",
       "      <td>4.241379</td>\n",
       "      <td>4.258621</td>\n",
       "      <td>4.120690</td>\n",
       "      <td>4.465517</td>\n",
       "      <td>0.241379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>341</td>\n",
       "      <td>17</td>\n",
       "      <td>4.352941</td>\n",
       "      <td>3.470588</td>\n",
       "      <td>2.294118</td>\n",
       "      <td>3.117647</td>\n",
       "      <td>3.117647</td>\n",
       "      <td>4.176471</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>102</td>\n",
       "      <td>61</td>\n",
       "      <td>3.573770</td>\n",
       "      <td>4.049180</td>\n",
       "      <td>2.688525</td>\n",
       "      <td>3.016393</td>\n",
       "      <td>2.934426</td>\n",
       "      <td>3.557377</td>\n",
       "      <td>1.147541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>15</td>\n",
       "      <td>71</td>\n",
       "      <td>4.718310</td>\n",
       "      <td>6.042254</td>\n",
       "      <td>5.140845</td>\n",
       "      <td>4.126761</td>\n",
       "      <td>5.422535</td>\n",
       "      <td>5.267606</td>\n",
       "      <td>2.267606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>195</td>\n",
       "      <td>45</td>\n",
       "      <td>5.933333</td>\n",
       "      <td>6.733333</td>\n",
       "      <td>6.800000</td>\n",
       "      <td>6.488889</td>\n",
       "      <td>6.777778</td>\n",
       "      <td>6.688889</td>\n",
       "      <td>2.022222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>315</td>\n",
       "      <td>6</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.166667</td>\n",
       "      <td>1.333333</td>\n",
       "      <td>4.166667</td>\n",
       "      <td>3.833333</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>376</td>\n",
       "      <td>29</td>\n",
       "      <td>3.827586</td>\n",
       "      <td>3.206897</td>\n",
       "      <td>2.103448</td>\n",
       "      <td>3.241379</td>\n",
       "      <td>3.103448</td>\n",
       "      <td>3.517241</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   community  members  fighting_skills  durability    energy  intelligence  \\\n",
       "0        570       10         4.300000    5.600000  4.800000      4.400000   \n",
       "1        451      173         3.225434    2.815029  2.075145      3.005780   \n",
       "2        356       58         4.551724    4.879310  4.241379      4.258621   \n",
       "3        341       17         4.352941    3.470588  2.294118      3.117647   \n",
       "4        102       61         3.573770    4.049180  2.688525      3.016393   \n",
       "5         15       71         4.718310    6.042254  5.140845      4.126761   \n",
       "6        195       45         5.933333    6.733333  6.800000      6.488889   \n",
       "7        315        6         4.000000    4.166667  1.333333      4.166667   \n",
       "8        376       29         3.827586    3.206897  2.103448      3.241379   \n",
       "\n",
       "      speed  strength    flight  \n",
       "0  5.100000  4.700000  0.000000  \n",
       "1  2.358382  2.502890  0.566474  \n",
       "2  4.120690  4.465517  0.241379  \n",
       "3  3.117647  4.176471  0.000000  \n",
       "4  2.934426  3.557377  1.147541  \n",
       "5  5.422535  5.267606  2.267606  \n",
       "6  6.777778  6.688889  2.022222  \n",
       "7  3.833333  4.000000  0.000000  \n",
       "8  3.103448  3.517241  0.000000  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_query(\"\"\"\n",
    "MATCH (c:Character)-[:HAS_STATS]->(stats)\n",
    "RETURN c.labelPropagation as community, count(*) as members, \n",
    "       avg(stats.fighting_skills) as fighting_skills,\n",
    "       avg(stats.durability) as durability,\n",
    "       avg(stats.energy) as energy,\n",
    "       avg(stats.intelligence) as intelligence,\n",
    "       avg(stats.speed) as speed,\n",
    "       avg(stats.strength) as strength,\n",
    "       avg(CASE WHEN c.flight = 'true' THEN 7.0 ELSE 0.0 END) as flight\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OUqOLjljSsIW"
   },
   "source": [
    "We can notice that the Label Propagation algorithm found twice as many communities as the Louvain Modularity algorithm. Some of them are relatively tiny. For example, the community with an id 395 has only three members, and all their average stats are at 1.0 value. They are the heroes that go by the name of Maggott, Deathbird, and Slayback. Funky names. The most powerful community has an id of 137 and only 23 members. Remember, the most powerful community found by the Louvain Modularity algorithm had 46 members and a slightly lower value of average stats.\n",
    "\n",
    "## Conclusion\n",
    "I hope you have learned some tricks on performing network analysis in Neo4j with the help of APOC and GDS libraries. There are still many things we could do with this graph, so expect a new post shortly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BcGLRfhVSsIW"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "name": "Exploratory graph analysis.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
